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SUMMARY 


Problem 


The  armed  services,  in  cooperation  with  the  Department  of  Defense,  have  undertaken 
a  joint  effort  to  investigate  the  feasibility  of  measuring  on-the-job  performance  and  using 
information  from  these  measures  to  set  military  enlistment  standards.  Before  the  linkage 
between  performance  and  enlistment  standards  can  be  attempted,  comprehensive  and 
economical  measures  of  job  performance  must  be  developed  and  validated. 

The  3ob  Performance  Measurement  (3PM)  program  is  developing  hands-on  and 
substitute  measures  of  job  performance  for  selected  ratings.  Prior  to  the  development  of 
new  measures  for  each  rating,  it  is  essential  to  examine  and  evaluate  existing  technology 
as  sources  of  job  performance  information. 

Objective 

The  objective  of  this  effort  was  to  determine  the  feasibility  of  utilizing  existing  Navy 
data  bases,  programs,  and  simulators  as  alternative  sources  of  job  performance  informa¬ 
tion. 

General  Approach 

Data  were  collected  by  reviewing  documentation  relating  to  each  system  and 
interviewing  cognizant  personnel.  One  data  base,  five  programs,  and  eight  simulators 
were  evaluated.  The  structure,  current  use,  and  possible  performance  measurement  uses 
were  examined  for  each  system. 

Conclusions 


The  examined  Navy  data  bases,  programs,  and  simulators  do  not  obviate  the  need  for 
a  performance  testing  program,  but  these  systems  can  support  such  a  program.  In 
selected  ratings,  existing  data  bases  may  eliminate  the  need  for  a  complete  job  analysis 
prior  to  development  of  performance  measures.  Programs  may  provide  items  for  testing 
personnel,  eliminating  the  need  for  time-consuming,  costly  item  development  in  all 
situations.  Simulators  may  be  used  as  substitute  measures  of  job  performance  with 
modification  of  software. 

Recommendations 


The  following  recommendations  are  made  on  the  basis  of  the  current  effort:  (1)  Navy 
Occupational  Task  Analysis  Program  (NOTAP)  questionnaires  can  be  used  to  determine 
the  representativeness  and  criticality  of  tasks  to  ratings;  (2)  Personnel  Qualification 
Standards  (PQS)  and  Propulsion  Examination  Board  (PEB)  items  should  be  examined  and 
adopted  whenever  appropriate;  (3)  extant  sources  of  technical  information,  such  as  the 
Engineering  Operational  Sequencing  System  (EOSS),  should  be  used;  (U)  the  Electronic 
Equipment  Maintenance  Trainer  (EEMT)  should  be  used  as  a  high-fidelity  substitute 
measure  of  hands-on  performance;  (5)  two  training  simulators  that  contain  actual 
equipment  (i.e.,  the  Practical  Application  Laboratory  and  the  Navy  Tactical  Data  System) 
should  be  used  to  collect  hands-on  performance  data;  and  (6)  analyses  of  additional  data 
bases,  programs,  and  simulators  should  be  conducted  to  form  a  broad  network  of  job 
performance  information  for  use  by  many  groups. 


CONTENTS 


Page 


INTRODUCTION  .  1 

Problem  . 1 

Objective  .  1 

Background  .  1 

APPROACH  .  2 

DATA  BASE  .  2 

Background  . 2 

Importance  to  the  3PM  Project  .  2 

Format  of  the  Data  Base  Examination  and  Evaluation  .  2 

Navy  Occupational  Task  Analysis  Program  (NOTAP) .  3 

Information  Description  . 3 

Availability .  4 

General  Remarks  .  4 

PROGRAMS  .  4 

Background  .  4 

Importance  to  the  3PM  Project  .  5 

Format  of  the  Program  Examination  and  Evaluation  .  5 

Personnel  Qualifications  Systems  (PQ$) .  6 

Information  Description  .  6 

Availability .  7 

General  Remarks  .  8 

Fleet  Feedback  System  (FFS) . .  9 

Information  Description  .  9 

Availability .  10 

General  Remarks  .  10 

Personnel  and  Training  Education  Program  (PTEP)  .  10 

Information  Description  .  10 

Availability . . .  11 

General  Remarks  .  11 

Propulsion  Examination  Board  (PEB)  . 12 

Information  Description  . 12 

Availability .  13 

General  Remarks  .  13 

TRAINING  SIMULATORS .  14 

Background  .  14 

Importance  to  the  3PM  Project  .  14 

Format  of  Training  Simulator  Examination  and  Evaluation  .  14 

AC  Two  Speed  Motor  Controller  Trainer  (1 1E15)  .  15 

Hardware  and  Software  .  15 

Output  . 15 

Fidelity  .  15 


Current  usage  . 

Additional  Uses  . . . 

Automated  Communications  Emulator  (ACE) 
Hardware  and  Software  . 


$ 


>11 


Current  Usage  . 

Additional  Uses  . . . 

Close  in  Weapons  System  Trainer  (I1G2) 
Hardware  and  Software  . 


Current  Usage  . . . 

Additional  Uses  . 

Cutler-Hammer  Logic  Controller  Elevator  Trainer 
Hardware  and  Software  . . . . 


Current  Usage  . 

Additional  Uses  . 

Electronic  Equipment  Maintenance  Trainer  (EEMT) 
Hardware  and  Software  . 


Current  Usage  . 

Additional  Uses  . . . 

Navy  Tactical  Data  System  Laboratory  (NTDS) 
Hardware  and  Software  . 


Current  Usage  . 

Additional  Uses  . 

Practical  Applications  Laboratory  (PRAC  DECK) 
Hardware  and  Software  . 


Current  Usage  . 

Additional  Uses  . 

Small  Craft  Electrical  System  Trainer  (1  IE  18) 
Hardware  and  Software  . . . . . 


Current  Usage  . . 
Additional  Uses  , 


RECOMMENDATIONS 

Data  Bases  . 

Programs . 

Simulators . 

CONCLUSIONS  . 


•Vi 


REFERENCES 


.  t\ 


»’>  r»  t' 


se\0'Ovfl'00*«»00  00  00  00  0«SM'4'4M'<J>40S5'9NONON9\^ 


INTRODUCTION 


Problem 


The  Navy  has  been  assigned  the  task  of  developing  reliable  and  valid  measures  of  job 
performance  (Laabs  <5c  Berry,  in  press).  The  Job  Performance  Measurement  (JPM) 
program  has  been  created  to  carry  out  this  task.  The  JPM  program  is  developing  hands-on 
job  performance  tests  and  substitute  measures  of  job  performance  for  selected  ratings. 
Ratings  were  selected  based  on  criticality  to  the  Navy's  mission  and  population  size.  The 
ratings  include:  (1)  machinist's  mate  (MM),  (2)  radioman  (RM),  (3)  electronics  technician 
(ET),  (4)  operations  specialist  (OS),  (5)  fire  control  technician  (FC),  and  (6)  electrician's 
mate  (EM).  Prior  to  developing  measures  for  each  of  the  selected  ratings,  an  effort  was 
completed  to  determine  the  existing  sources  of  Navy  performance  data. 

The  current  work  examines  alternative  sources  of  job  performance  information.  The 
Navy  currently  uses  data  bases,  programs,  and  simulators  for  a  variety  of  purposes.  Each 
of  these  systems  contains  a  great  deal  of  information.  In  order  to  avoid  duplication  of 
effort  and  to  use  Navy  resources  to  their  fullest,  several  systems  have  been  examined  and 
evaluated.  This  allows  for  the  most  cost  effective  use  of  time  and  money. 

The  following  discussions  of  data  bases  and  programs  contain  information  that  is 
applicable  to  many  ratings.  The  section  on  training  simulators  contains  information  that 
is  pertinent  to  the  JPM  program's  selected  ratings.  That  is,  simulators  that  are  currently 
used  by  the  six  selected  ratings  were  examined. 

Although  the  initial  review  of  materials  was  performed  for  use  by  the  JPM  program, 
the  information  obtained  may  be  used  by  any  group  developing  training  programs,  job 
performance  criteria,  or  selection  predictors. 

Objective 

The  objective  is  to  identify,  describe,  and  provide  an  initial  evaluation  of  alternative 
sources  of  job  performance  information,  including  Navy  data  bases,  programs,  and 
simulators. 

Background 

The  Armed  Services,  in  cooperation  with  the  Department  of  Defense,  have  under¬ 
taken  a  joint  effort  to  investigate  the  feasibility  of  measuring  on-the-job  performance  and 
using  these  measures  to  set  military  enlistment  standards.  Before  the  linkage  between 
performance  and  enlistment  standards  can  be  attempted,  comprehensive  and  economical 
measures  of  job  performance  must  be  developed  and  validated. 

The  Navy  is  developing  hands-on  and  substitute  measures  of  job  performance.  The 
Navy  currently  holds  a  great  deal  of  information  that  relates  to  job  performance.  Prior  to 
developing  performance  measures,  it  is  essential  to  evaluate  existing  technology.  This 
work  serves  as  a  first  step  in  the  JPM  program  by  identifying,  describing,  and  evaluating 
existing  data  bases,  programs,  and  simulators  that  have  potential  for  use  in  performance 
measurement. 


APPROACH 


The  approach  to  this  effort  included: 

1.  Identification  of  potential  sources  through  interviews  with  Navy  management 
personnel  and  a  perusal  of  the  Navy  Directives  checklist  (5200). 

2.  Assembling  available  documentation  for  each  potential  source. 

3.  Conducting  interviews  with  cognizant  personnel  where  necessary  to  augment  the 
printed  information. 

k.  Evaluating  the  existing  sources  for  their  potential  to  provide  job  performance 
information  or  otherwise  to  support  the  Navy  3PM  program. 


DATA  BASE 


Background 

The  need  for  a  comprehensive  Navy  occupational  information  system  has  been 
recognized.  The  goal  of  such  a  system  is  to  provide  a  proficient  occupant  for  every  billet 
in  the  Fleet.  The  foundation  of  such  a  system  would  be  an  occupational  data  base 
containing  the  identification  of  jobs,  duties,  skills,  and  tasks  for  each  rating.  The  primary 
occupational  data  base  currently  available  for  this  purpose  is  the  Navy  Occupational  Task 
Analysis  Program  (NOTAP).  NOTAP  data  are  used  as  the  basis  of  occupational  standards 
(OCCSTDs)  and  Navy  Enlisted  Classifications  (NECs).  OCCSTDs  define  the  enlisted  tasks 
required  of,  and  within,  specified  occupational  entities  (rates  and  ratings).  They  are 
minimum  standards  which  represent  the  lowest  level  of  responsibility  to  fulfill  Navy 
needs.  NECs  identify  skills  requiring  more  specific  identification  than  is  provided  by 
rates  and  ratings  and  are  not  rating-wide  requirements  (Department  of  the  Navy,  1975). 

Importance  to  the  3PM  Project 

Occupational  data  bases  ideally  provide  detailed,  comprehensive  information  regard¬ 
ing  Navy  occupations.  OCCSTDs,  training  materials  and  curricula,  and  other  personnel 
functions  all  require  specific  job  task  data  that  can  be  used  in  the  preparation  of  job 
performance  evaluation  and  job  training  programs.  The  data  base  of  NOTAP  is  examined 
for  its  potential  usefulness  in  the  performance  measurement  area. 

Format  of  the  Data  Base  Examination  and  Evaluation 


The  data  base  examined  in  this  report  is  that  associated  with  NOTAP.  The  following 
format  was  used. 

1.  Information  Description 

a.  Type  of  Information 

b.  Data  Elements 

c.  Source  of  Information 

d.  Currency  and  Maintenance 


2.  Availability 

a.  Normal  Usage 

b.  Access  Procedures 

3.  General  Remarks 

Section  1,  Information  Description,  provides  an  overview  of  the  data  base.  The 
nature  of  the  information  (a),  for  example,  task  inventory,  is  described  first.  Individual 
elements  comprising  the  system  (b),  are  then  described.  Source  of  information  (c)  refers 
to  the  manner  in  which  the  information  was  obtained  and  what  resources  were  used. 
Currency  and  maintenance  (d)  discusses  the  currency  and  upkeep  of  the  information  kept 
in  the  data  base. 

Section  2,  Availability,  discusses  (a)  how  the  data  base  is  being  used  currently  and  (b) 
how  the  information  can  be  accessed.  General  Remarks,  Section  3,  includes  advantages, 
limitations,  and  a  short  evaluation  of  the  data  base's  potential  for  providing  job 
performance  information  to  the  3PM  project. 

Navy  Occupational  Task  Analysis  Program  (NOTAP) 

Information  Description 

The  Navy  Occupational  Development  and  Analysis  Center  (NODAC),  a  detachment  of 
the  Naval  Military  Personnel  Command  (NMPC-5),  is  the  primary  source  of  job  informa¬ 
tion  on  enlisted  ratings  for  the  Navy.  The  NOTAP  is  a  program  conducted  by  NODAC 
which  surveys  the  work  performed  by  personnel  in  each  Navy  rating.1 

Type  of  Information.  NODAC  designs  and  administers  task  inventories  that  contain 
detailed  task  statements.  An  inventory  typically  contains  several  hundred  task  state¬ 
ments  for  Navy  enlisted  jobs  (Robertson,  1984).  Other  sections  of  the  inventory  contain 
items  on  watchstanding  and  collateral  duties,  tools  and  equipment,  and  types  of  ships  or 
aircraft.  A  task  inventory  is  developed  for  each  rating  or  occupational  area  surveyed  by 
NOTAP. 

The  inventory  is  a  composite  of  the  tasks  and  supporting  background  material 
applicable  to  the  survey  population  and  is  assembled  from  on-site  observations,  inter¬ 
views,  review  of  publications  and  directives,  and  assistance  of  subject  matter  experts 
(SMEs).  Inventory  development  comprises  the  following: 

I.  A  tentative  task  list  is  developed.  Sources  of  information  include  current 
inventories,  jury  of  experts,  and  developers  of  new  hardware  and  systems. 


2.  The  task  list  is  authenticated  by  rating  advisors,  warfare  sponsors,  enlisted 
community  managers,  training  commands,  a  group  of  SMEs,  and  job  incumbents. 

3.  3ob  incumbents  verify  the  performance  of  the  tasks  on  the  list  and  may  also 
provide  descriptive  characteristics  of  the  tasks. 


JThe  Navy  Enlisted  Professional  Development  Information  System  (NEPDIS)  was 
examined  as  a  potential  source  of  job  performance  information.  This  experimental 
program  was  designed  to  be  a  fully  computerized  information  assembly  and  analysis 
system  that  would  support  manpower,  personnel,  and  training  management  (Ansbro,  1982). 
NEPDIS  has  not  progressed  beyond  experimental  status  and  is  not  currently  is  use; 
therefore,  it  has  not  been  included  in  this  report. 


Data  Elements.  Data  collected  include: 
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1.  Conditions  under  which  the  tasks  are  performed. 


2.  Cues  that  initiate  and  guide  performance  of  the  task. 


3.  Standards  which  represent  adequate  task  performance. 


k.  Task  elements  (those  parts  of  the  task  necessary  to  its  performance  but  are  not 
performed  as  an  end  in  themselves). 


Job  task  data  are  analyzed  to  provide  frequency  distributions  of  the  tasks  for  a 
profile  of  job  activities  unique  to  the  rating,  to  each  paygrade,  and,  as  needed,  to 
particular  types  of  ships,  submarines,  or  aircraft.  These  frequency  distributions  provide  a 
principal  source  of  data  for  the  development  of  OCCSTDs  and  course  curricula.  NOTAP 
data2  include  all  ratings  at  paygrades  E-4  through  E-9. 


Currency  and  Maintenance.  About  16  ratings  are  surveyed  each  year.3  High- 
technology  ratings,  such  as  ETs,  are  updated  every  3  years.  Low  technology  ratings,  such 
as  storekeeper,  are  surveyed  every  6  years.  NODAC  is  responsible  for  maintaining  the 
occupational  data  bank  for  the  Chief  of  Naval  Personnel. 


Availability 


Normal  Usage.  NOTAP  data  are  used  to  develop  and  validate  OCCSTDs,  training 
materials  and  curricula,  and,  along  with  data  from  other  programs,  help  to  validate  Navy 
billet  requirements. 


Access  Procedures.  NOTAP  data  may  be  obtained  from  the  Officer  in  Charge, 
NODAC. 


General  Remarks 


NOTAP  data  are  useful  for  obtaining  a  comprehensive  picture  of  an  occupation. 
SMEs  are  needed  to  add  more  specific  information  concerning  individual  tasks  and  to 
determine  on  which  tasks  a  performance  test  should  focus. 


PROGRAMS 


Background 


Various  Navy  programs  have  been  developed  over  the  years  to  meet  the  growing 
demands  of  training  school  curricula,  on-the-job  training,  and  performance  evaluation. 
They  take  on  various  forms  ranging  from  very  focused  and  training-specific  guidance 
programs  to  extremely  broad  and  subjective  evaluation  programs. 


2,3 D.  Shotts,  personal  communication,  August  13,  1986. 


Each  program  plays  a  different  role  in  the  overall  functioning  of  the  Navy  personnel 
system,  and  the  scope  varies  from  structuring  training  to  evaluating  performance.  For 
example,  specific  performance  evaluation  of  the  boiler  technician  (BT)  and  MM  ratings  is 
accomplished  by  the  Propulsion  Examining  Board  (PEB).  In  contrast,  the  Fleet  Feedback 
System  (FFS)  evaluates  training  school  effectiveness  for  all  technical  ratings.  Each 
program  serves  a  well-defined  purpose  in  ensuring  that  personnel  are  qualified  to  do  their 
jobs. 

Importance  to  the  3PM  Project 

Potential  importance  of  existing  programs  is  twofold:  (1)  to  provide  actual  data  so 
that  new  measures  will  not  have  to  be  constructed  and  (2)  to  provide  instruments  for 
measuring  job  performance. 

There  are  advantages  in  reviewing  existing  programs  for  usable  instruments  to 
supplement  those  being  constructed.  First,  time  is  saved.  Some  of  the  instruments  are 
already  complete.  Second,  costs  are  kept  down.  Needless  duplication  of  effort  is  costly 
in  terms  of  personnel  and  resources.  Third,  procedures  that  work  and  do  not  work  are 
already  known  in  many  cases.  Direct  use  can  be  made  of  program  elements  that  have 
proven  valid  and  reliable. 

Format  of  the  Program  Examination  and  Evaluation 
Four  programs  are  described  herein: 

1.  Personnel  Qualification  Standards  (PQS). 

2.  Fleet  Feedback  System  (FFS). 

3.  Personnel  and  Training  Evaluation  Program  (PTEP). 

4.  Propulsion  Examination  Board  (PEB). 

The  Program  Evaluation  used  the  following  format: 

1.  Information  Description 

a.  Type  of  Information 

b.  Data  Elements 

c.  Source  of  Information 

d.  Currency  and  Maintenance 

2.  Availability 

a.  Normal  Usage 

b.  Access  Procedures 

3.  General  Remarks 


Section  1  contains  a  description  of  the  information  comprising  each  program  and  its 
primary  purposes.  Type  of  Information  (a),  refers  to  the  format  each  program  utilizes  in 
order  to  carry  out  its  appointed  task,  for  example,  oral  interview.  Data  elements  (b) 
describe  specific  program  components,  such  as  sets  of  questions  administered  to  trainees. 


Source  of  Information  (c)  describes  how  the  material  was  derived  and  where  it  was 
developed.  Currency  and  Maintenance  (d)  contains  information  regarding  how  the 
program  is  kept  up  to  date  with  changing  technology  and  requirements. 


Section  2  describes  (a)  how  and  by  whom  the  program  is  currently  being  used  and  (b) 
how  the  information  can  be  accessed.  Section  3,  General  Remarks,  contains  a  brief 
assessment  of  advantages,  limitations,  and  potential  uses  of  the  program  for  use  in  the 
3PM  project. 

Personnel  Qualifications  System  (PQS) 

Information  Description 

The  PQS  is  a  system  designed  to  standardize  and  formalize  enlisted  on-the-job 
training.  The  PQS  is  a  training  guide  that  provides  the  minimum  knowledge  and  skill 
requirements  a  person  must  have  in  order  to  qualify  for  a  particular  job  or  watchstation.'* 

Type  of  Information.  The  PQS  is  divided  into  three  sections.  Section  100 
(fundamentals)  contains  the  fundamental  knowledge  or  "book  learning"  necessary  for 
satisfactory  understanding  of  the  watchstation  duties.  Section  200  (systems)  is  designed 
to  acquaint  the  watchstander  with  the  systems  he  or  she  will  be  required  to  operate  at  the 
watchstation.  Section  300  (watchstations)  lists  the  tasks  required  to  satisfactorily 
perform  in  order  to  achieve  final  PQS  qualification  for  a  particular  watchstation. 

The  PQS  program  consists  of  the  standard  booklet  and  the  progress  chart,  or  any 
record  (e.g.,  spread  sheet)  of  final  results  as  determined  by  the  type  commander. 
Standard  booklets  are  developed  that  contain  knowledge  questions  and  hands-on  perfor¬ 
mance  requirements  that  are  specific  to  a  watchstation  and  associated  system  and 
equipment. 

Data  Elements.  The  standard  booklet  contains  questions  that  the  trainee  must  be 
able  to  answer  and  performance  items  which  he  or  she  must  be  able  to  do  in  order  to 
qualify  for  a  particular  watchstation  or  workstation.  Answer  books  containing  line-item 
short  answers  are  available  for  Section  100  (fundamentals)  and  Section  200  (systems)  for 
some  PQS  watchstations. 

Source  of  Information.  The  requirements  set  forth  in  the  PQS  packages  are 
developed  by  SMEs  who  work  with  the  PQS  Development  Group  located  at  the  Naval 
Training  Center  (NTC),  San  Diego,  California.  The  Development  Group  consists  of  four 
departments:  operations,  aviation,  engineering,  and  weapons.  The  SMEs,  who  are  current 
job  incumbents,  meet  for  2-week  workshops  to  produce  the  PQS.  Standards  are  written  to 
include  everything  one  needs  to  know  in  order  to  perform  successfully  at  a  given 
watchstation. 


**The  Shipboard  Propulsion  Plant  Operator  Training  (SPPOT)  program  is  a  research 
and  development  effort  that  incorporates  PQS  in  order  to  provide  on-the-job  training  aids 
for  BTs  and  MMs  (Main,  Abrams,  Chiles,  Todd,  &  Cunanan,  1981).  SPPOT  development 
has  been  terminated  and  the  program  is  being  replaced  by  the  Propulsion  Plant  Training 
Manual.  Therefore,  it  has  not  been  included  in  this  report. 

*3.  Weaver,  personal  communication,  August  7,  1986. 


Currency  and  Maintenance.  There  are  approximately  700  PQSs  now  in  use.  These  are 
revised  about  every  3  years.6  The  PQS  procedure  is  applied  primarily  on  the  basis  of 
requests  due  to  Navy  training  requirements  and  Fleet  requirements.  Coverage  of  both  old 
and  new  systems,  equipment,  and  tasks  may  not  be,  therefore,  comprehensive.  PQS  is  not 
applicable  to  nuclear  propulsion  or  the  Fleet  ballistic  missile  weapons  systems.  Ratings 
that  have  no  associated  PQS  include  personnelman,  yeoman,  and  dental  technician. 

Recent  years  have  seen  a  shift  away  from  maintenance-oriented  programs  toward 
operator-oriented  programs.  Design  of  PQS  has  become  more  in-depth;  sections  that  were 
once  optional,  such  as  safety  precautions  unique  to  a  particular  system,  are  now  required. 
The  job  qualification  requirement  OQR)  is  a  locally  developed  PQS  designed  specifically 
for  a  local  need.  The  new  PQS  is  submitted  for  approval  and  can  then  be  used  command 
wide. 

A  PQS  manager's  guide  has  been  created  to  provide  guidance  for  PQS  implementation 
and  management.  It  is  designed  for  use  by  supervisors  and  managers  who  are  assigned  the 
responsibility  of  administering  the  PQS  program,  and  it  assists  in  overall  management, 
record  keeping,  and  inspection  guidelines  (Office  of  Naval  Education  and  Training,  1986). 

Availability 

Normal  Usage.  PQSs  are  written  for  watchstations,  not  for  equipment  or  ratings. 
Any  trainee  at  a  given  watchstation,  regardless  of  rating,  must  complete  the  relevant 
PQS. 

PQS  qualifiers  are  normally  E-5  or  above  and  ideally  are  acknowledged  experts  in  a 
specified  area  of  qualification.  Qualifiers  serve  a  dual  role  in  exercising  quality  control 
over  the  PQS  program  while  also  serving  as  a  point  of  reference  to  assist  the  trainees  in 
acquiring  the  knowledge  and  skills  they  have  not  achieved  on  their  own. 

The  PQS  begins  with  a  fundamentals  section  covering  the  basic  knowledge  and 
principles  needed  to  understand  the  equipment  or  duties  to  be  studied.  This  knowledge  is 
normally  acquired  during  the  school  phase  of  training.  References  are  listed  at  the 
beginning  of  each  fundamentals  section  for  the  individual  who  has  not  been  to  school  or 
needs  a  refresher. 

Fundamentals  the  individual  will  have  to  complete  are  listed  in  each  watchstation 
(Section  300).  The  trainee  completes  all  required  fundamentals  before  starting  the 
systems  and  watchstation  portions  of  the  PQS.  Individuals  contact  their  qualifiers  when 
their  understanding  of  one  or  more  fundamentals  is  complete.  All  line  items  in  the 
fundamentals  section  must  be  satisfactorily  answered  before  the  section  can  be  signed 
off.  For  individuals  who  have  been  to  school  or  are  requalifying,  the  qualifier  may  ask 
representative  line  items  to  determine  if  the  necessary  knowledge  for  the  watchstation 
has  been  retained.  If  the  command  requires  an  oral  board  or  written  examination  for  final 
qualification,  any  questions  from  the  fundamentals  required  by  the  watchstation  may  be 
asked. 


SG.  Houston,  personal  communication,  August  7,  1986. 


Access  Procedures.  Job  task  information  is  readily  available  in  the  standard 
booklets.  Task-by-task  qualification  data  are  not  retained  in  a  person's  service  record. 
The  only  information  retained  is  whether  or  not  a  person  is  qualified  for  a  watchstation 
and  the  date  of  the  decision. 

General  Remarks 

PQS  provides  information  about  the  basic  knowledge  and  principles  in  a  specified  area 
of  qualification.  The  questions  are  potentially  useful  for  assessing  job  performance. 
However,  the  use  of  the  existing  instrument  is  not  recommended.  There  are  several 
problems  with  using  PQS  examination  performance  as  measures  of  job  performance: 

1.  The  evaluation  process  is  not  standardized.  Lack  of  specific  evaluation 
guidelines  makes  it  extremely  unlikely  that  for  any  given  knowledge  or  skill,  different 
raters  will  use  the  same  criteria  when  judging  performance.  Use  of  different  criteria 
necessitates  a  lack  of  reliability.  The  system,  therefore,  does  not  appear  to  provide 
performance  information  related  to  individuals  and  teams  that  is  objective,  valid,  or 
reliable. 

2.  Supervisors  differ  in  how  much  they  emphasize  the  PQS.  Inaccurate  data  may  be 
collected  or  records  may  be  poorly  maintained.  Supervisors  are  responsible  for  the 
accuracy,  updating,  and  tailoring  of  the  PQS  to  fit  their  command's  needs.  As  a  result, 
PQS  is  not  standard  across  commands.  Supervisors  are  also  responsible  for  initiating 
feedback  to  the  PQS  Development  Group;  therefore,  the  validity  of  the  system  depends 
partly  on  the  participation  of  supervisors. 

3.  The  dichotomous  nature  of  the  performance  scores  on  PQS  tells  nothing  of  the 
quality  of  performance.  PQS  includes  no  scales  by  which  to  indicate  how  well  the  trainees 
answer  the  questions  or  perform  the  tasks.  The  information  collected  with  the  PQS 
indicates  only  whether  the  trainee  did  or  did  not  achieve  the  requirements. 

4.  Differential  treatment  may  be  given  those  trainees  who  attend  "A"  school  over 
those  who  do  not.  Qualification  petty  officers  may  assume  that  graduates  of  "A"  school 
are  knowledgeable  about  the  material  in  the  fundamentals  section  and  give  the  graduate 
credit  without  formal  examination. 

5.  The  rate  of  qualifying  differs  widely  across  commands.  Some  supervisors  push 
for  rapid  qualification,  while  others  allow  personnel  to  qualify  at  a  slower  pace. 
Opportunity  to  complete  the  PQS  may  differ  by  squadron.  Squadron's  missions  vary  in 
complexity  and  difficulty,  affecting  the  amount  of  time  a  trainee  can  devote  to  PQS. 
Rate  of  qualifying  can  also  be  slowed  by  the  availability  of  the  qualification  petty  officers 
for  assisting  trainees  in  using  the  reference  material  and  checking  off  achievements. 

The  requirements  in  the  standard  booklets  represent  a  critical  task  analysis  for  a 
particular  watchstation  and  its  associated  system  and  equipment.  Fundamentals  (Section 
100),  systems  (Section  200),  and  watchstations/workstations  (Section  300)  are  possible 
sources  for  development  of  performance  measures.  Each  of  these  sections  contains 
examples  of  possible  performance  test  questions. 

No  estimate  of  the  task's  importance  or  criticality  to  the  job  is  offered.  In  addition, 
the  standard  booklets  often  do  not  specify  the  steps  required  for  task  completion.  In 
order  to  use  the  information  from  the  standard  booklets  for  test  construction,  more 
detailed  task  analyses  must  be  performed  to  determine  the  criticality  of  tasks  and  the 
steps  required  for  task  completion. 


Fleet  Feedback  System  (FFS) 


Information  Description 

The  FFS  is  operated  by  the  Naval  Education  and  Training  Command  (NAVEDTRA- 
COM)  to  provide  external  evaluation  of  training  programs.  It  is  based  on  unsolicited 
reports  and  on  sample  surveys  that  ask  about  the  adequacy  of  the  training  in  a  given 
course  and  the  importance  of  this  training  to  the  particular  ship  or  activity.  A  report  of 
the  external  evaluation  findings  are  forwarded  to  the  commanding  officers  of  the 
appropriate  service  school  commands  for  action. 

Purposes  of  the  FFS  include  (1)  product  evaluation,  in  order  to  determine  whether  or 
not  formal  training  is  effective  in  the  Fleet  and  responsive  to  Fleet  needs;  (2) 
identification  of  problems,  and  (3)  support  of  improvement  and  contribution  to  readiness 
in  the  Fleet.  The  program  is  intended  to  improve  the  quality  of  individual  training  to 
assure  a  beneficial  impact  in  improving  overall  Fleet  readiness  without  imposing  an 
administrative  burden  on  the  Fleet  (Office  of  the  Chief  of  Naval  Operations,  19S1). 

Type  of  Information.  Feedback  on  training  adequacy  is  solicited  and  is  based  on 
observed  performance  of  graduates,  performance  in  the  Fleet  setting,  and  actual 
performance  as  compared  to  course  objectives.  Feedback  regarding  the  appropriateness 
of  objectives  is  sought  to  determine  if  objectives  are  responsive  to  Fleet  needs,  required 
or  important,  and  adequate  in  scope. 

The  training  appraisal  and  feedback  system  provides  the  following  methods  of 
training  effectiveness  feedback  to  NAVEDTRACOM  from  commands  receiving  graduates 
of  specialized  skill  training  courses  of  instruction: 

Level  1:  Unsolicited  feedback  from  any  source  as  to  the  adequacy  and  effectiveness 
of  training.  Training  inadequacy  reports  are  available  to  provide  specific  and  useful 
feedback  from  which  the  need  for  training  program  changes  may  be  identified. 

Level  2:  Feedback  provided  through  questionnaires  that  are  centrally  administered 
by  CNET.  Questionnaires  are  organized  by  training  activity  and  cover  the  sponsor's 
training  requirements  and  terminal  objectives.  Questionnaires  employ  rating  scales  and 
comments.  Assessments  are  made  by  immediate  supervisors  on  samples  of  specialized 
skill  training  course  graduates. 

Level  3:  Feedback  obtained  by  in-depth  analysis  of  training  deficiencies  identified 
through  Levels  1  and  2  feedback.  Level  3  feedback  is  used  only  in  cases  in  which  required 
corrective  actions  cannot  be  ascertained  from  available  data.  Techniques  such  as 
structured  interviews  are  used  at  this  level. 

Data  Elements.  Skills  profiles,  which  comprise  the  Catalog  of  Navy  Training  Courses 
(CANTRAC),  are  used  to  construct  Level  2  questionnaires.  These  are  sent  to  supervisors 
of  training  school  graduates  who  compare  the  objectives  with  actual  job  performance. 
The  resulting  data  are  processed  by  the  Course  Curriculum  Model  Manager  and  statistical 
analyses  are  performed.  Items  are  concurrently  validated. 


Source  of  Information.  Level  2  questionnaires  are  constructed  from  skills  profiles. 
All  Navy  training  courses  are  required  to  utilize  skills  profiles  unless  an  external 


evaluation  system  has  been  incorporated  (e.g.,  construction).7  Responses  to  the  question¬ 
naires  are  inherently  subjective  because  supervisors  determine  by  their  own  judgment 
whether  or  not  objectives  have  been  met. 


Currency  and  Maintenance.  Level  2  surveys  are  scheduled  every  3  years  or  on  an  as- 
needed  basis.  Skills  profiles  are  kept  current  and  relevant  to  Navy  training  courses 
through  regular,  required  updating.  Schoolhouse  needs  fluctuate  with  budgetary  con¬ 
straints  that  may  result  in  reduced  training.  Research  and  development,  however,  provide 
ongoing  training  improvement  and  efficacy. 

Availability 

Normal  Usage.  The  FFS's  primary  functions  are  to  monitor  the  adequacy  of  training 
of  graduates,  improve  instruction  and  curriculum,  adjust  objectives,  and  evaluate  cost 
effective  alternatives. 

Access  Procedures.  The  skills  profiles  are  readily  available  on  microfische 
CANT  RAC.  Specific  objectives  and  training  course  material  can  be  assessed  by  using  the 
Course  Identification  Number  (CIN)  found  on  the  Skills  Profile.  Training  course  feedback 
data  are  not  available. 

General  Remarks 


The  surveys  are  essentially  customer  satisfaction  questionnaires  and  do  not  assess 
graduate  performance  on  the  job.  The  degree  to  which  a  course  meets  Chief  of  Naval 
Education  and  Training  (CNET)  goals  is  measured.  Individual  skills  profiles  are  compared 
to  supervisory  evaluations  for  groups  of  individual  participants  in  a  course.  This  may  be 
considered  a  measure  of  transfer  of  training;  it  is  not  a  measure  of  overall  job 
performance.  Performance  tests  are  not  used  at  any  of  the  three  levels  of  feedback. 
Therefore,  the  surveys  do  not  provide  useful  information  for  the  3PM  program. 

Personnel  and  Training  Education  Program  (PTEP) 

Information  Description 

The  PTEP  is  operated  for  the  Fleet  Ballistic  Missile  Weapon  System  ratings  T  C,  ET, 
and  missile  technician  (MT).  Its  purposes  are  to: 

1.  Provide  the  means  to  assess  the  quality  of  personnel  knowledge  and  skill  levels 
for  enlisted  personnel. 

2.  Provide  reports  to  commanding  officers  and  type  commanders  to  assist  in 
planning  training  and  determining  the  most  effective  use  of  personnel. 

Type  of  Information.  The  PTEP  has  developed  System  Achievement  Tests  (SATs)  for 
the  above  ratings.  The  SAT  is  a  criterion-referenced  instrument  based  on  test  design 
specifications  for  the  Atlantic  and  Pacific  Fleets  derived  from  Fleet  technician  (job 
incumbent)  input. 


7T.  Warwick,  personal  communication,  3une  19,  1986. 


The  PTEP  conducts  a  back-fitted  task  analysis  of  current  functions  during  a  1-week 
workshop.  Test  design  specifications  are  the  single  basis  for  selecting  test  items  for  each 
SAT  version.  Each  specification  contains  four  parts:  (1)  administration  and  security,  (2) 
watch  qualifications,  (3)  casualty  procedures,  and  (4)  equipment.  One  set  of  test  design 
specifications  is  provided  for  each  supervisor  and  watchstander  technician  rating. 

Data  Elements.  Tests  built  from  design  specifications  contain  up  to  360  items. 
Pass/fail  criteria  for  the  tests  are  set  at  a  cut  score  workshop  for  areas  and  subareas. 
The  Fleet  provides  job  incumbents  for  cut-score  workshops  where  these  minimum 
performance  levels  are  set.  Individual  and  group  data  are  maintained  and  reported  to 
nuclear-powered  Fleet  ballistic  missile  submarine  (SSBN)  and  type  commanders. 

Source  of  Information.  The  test  design  specifications  are  the  result  of  a  compre¬ 
hensive  task  analysis  for  each  rating.  Input  from  job  incumbents  provides  the  basis  for  all 
SAT  construction. 

Currency  and  Maintenance.  Routine  analyses  of  examinations  are  conducted  to 
ensure  validity  and  to  correct  errors.  Tests  are  replaced  every  6  months  with  an 
approximate  30  percent  change  in  test  items.  Forty-eight  new  tests  are  generated  each 
year.* 

Availability 

Normal  Usage.  A  self-contained,  boxed  set  of  tests,  answer  sheets,  pencils,  and  a 
proctor  guide  are  delivered  to  each  submarine  about  1  month  before  sailing.  Tests  are 
included  for  each  of  the  five  NEC  supervisors  and  watchstanders.  Test  security  is  the 
same  as  for  Advancement-in-Rate  examinations.  Each  ship  also  receives  a  Personnel 
Data  Sheet  (PDS)  for  each  crew  member  including  scores  on  previous  SATs,  number  of 
patrols,  and  records  of  all  formal  training  courses.  Individuals  update  their  own  PDS  and 
return  it  to  the  PTEP  along  with  their  test  answer  sheet.  The  PTEP  maintains  records  on 
all  strategic  weapons  systems  (S WS)  technicians. 

A  personnel  data  summary  report  by  ship  is  produced  quarterly.  These  reports  are 
provided  to  type  commanders,  NMPC,  and  the  Strategic  Systems  Program  Office  (SSPO). 
NMPC  uses  the  data  to  aid  in  personnel  assignments,  and  type  commanders  use  the  data  to 
monitor  crews  and  make  equitable  assignments.  Each  ship  commander  is  given  an 
individual  scoring  report  of  their  crew's  performance.  A  list  of  available  on  board  training 
materials  is  attached  to  the  scoring  report. 

Access  Procedures.  Test  design  specifications  and  performance  data  are  not 
available.  Only  SP-15  has  the  authority  to  direct  the  release  of  data.  Information 
regarding  PTEP  may  be  obtained  from  SSPO- 154,  Washington,  DC. 


General  Remarks 

SATs  are  already  in  use  for  the  FC,  ET,  and  MT  ratings. 

Although  the  PTEP  has  potential  for  use  in  the  development  of  job  performance 
measures,  the  data  are  not  readily  available.  There  are  several  advantages  to  the  PTEP. 

*E.  Hobson,  persona!  communication,  July  10,  1986. 


First,  the  PTEP  is  based  on  a  thorough  task  analysis.  Second,  the  PTEP  is  criteria-based. 
Standards  of  performance  and  criteria  are  set  and  performance  is  measured  against  these 
criteria.  Third,  the  PTEP  is  frequently  updated.  Tests  are  revised  every  6  months. 


The  PTEP  is  limited  in  to  potential  use  by  a  small  number  of  ratings.  In  addition,  it 
does  not  offer  at  this  time  a  hands-on  job  performance  test.  The  PTEP  tests  may  provide, 
however,  a  suitable  paper-and-pencil  substitute  for  hands-on  job  performance. 

Propulsion  Examination  Board  (PEB) 

Information  Description 

The  PEB  is  a  quality-control  agency  whose  final  responsibility  is  certifying  that  a 
ship's  steam  propulsion  plant  is  safe  to  steam  and  that  the  watch  teams  are  qualified.  The 
PEB  conducts  scheduled  ship  reviews  which  involve  a  Light-Off  Exam  (LOE)  and  an 
Operations/Propulsion  Plant  Exam  (OPPE).  The  objectives  of  these  exams  are  to  ensure 
that  the  material  readiness  and  safety  of  the  propulsion  plant  is  maintained,  the  level  of 
job  knowledge  is  adequate,  and  the  engineering  administrative  program  is  properly 
conducted. 

Type  of  Information.  The  PEB  uses  three  methods  to  determine  propulsion  plant 
personnel  readiness:  written  examinations,  oral  interviews,  and  watchstander  perfor¬ 
mance.  Written  tests  are  generated  by  computer  from  a  3,000  question  data  bank  and  are 
primarily  used  by  ships  to  evaluate  level  of  knowledge  and  training  effectiveness.  Oral 
interviews  are  used  to  establish  the  interviewee's  depth  of  understanding  and  allow 
observation  of  his  reaction  to  situations  that  test  judgment.  Watchstander  performance  is 
assessed  by  observing  a  team's  conduct  of  material  safety  checks,  routine  plant  evolu¬ 
tions,  and  casualty  control  exercises. 

Data  Elements.  The  PEB  maintains  a  question  bank  of  3,000  multiple  choice  and 
essay  items.’  Each  question  is  coded  according  to  (1)  type  of  propulsion  plant  (690#, 
1200//,  gas,  or  diesel),  (2)  watchstation,  and  (3)  ship  type.  Computer  programs  generate 
tests  by  selecting  questions  at  random  from  the  question  bank  according  to  the 
appropriate  code(s).  No  two  tests  are  alike.  Tests  are  created  for  watchstations,  not 
ratings.  Ratings  most  frequently  tested,  however,  are  BT  and  MM.  Th;  PEB  project 
officer  in  charge  of  an  individual  ship's  test  informs  the  PEB  testmake1  s  of  situations 
unique  to  that  ship  so  that  questions  are  relevant  and  pertinent. 

Source  of  Information.  Written  test  questions  are  composed  by  SMEs  and  PEB 
members.  Oral  interviews  are  conducted  by  an  expert  examiner.  The  examiner  conducts 
oral  group  interviews  by  asking  objective  questions  determined  to  be  relevant  to  a  given 
piece  of  equipment,  system,  or  procedure.  Answers  are  subjectively  evaluated,  and  the 
group  is  given  a  pass  or  fail  rating. 

The  watch  station  section  (300)  of  the  PQS  is  the  source  of  tasks  to  be  demonstrated 
and  evaluated  during  the  watchstander  task  performance  phase  of  the  examination.  Drill 
scenarios  are  based  on  the  Engineering  Operations  Casualty  Control  (EOCC). 


*B.  Davis,  personal  communication,  June  23,  1986. 


Currency  and  Maintenance.  Twice  a  year,  questions  are  added  to  the  question  pool. 
The  examination  coordinator  reviews  completed  test  items  periodically  and  may  delete 
items  that  are  consistently  missed. 

Availability 

Normal  Usage.  An  LOE  is  conducted  for  any  ship  spending  120  days  or  more  in  the 
yard.  The  LOE  consists  of  an  evaluation  of  four  major  areas:  material  readiness,  level  of 
knowledge,  engineering  department  administration,  and  fire  fighting  capability  (Depart¬ 
ment  of  the  Nav;  ^a).  OPPEs  are  scheduled  within  6  months  after  an  evolution 
requiring  an  LOE.  ->erational  Propulsion  Plant  Recertification  Examination  (OPPRE) 

is  scheduled  withir  months  after  an  OPPE  and  may  occur  unannounced.  OPPEs  do 

not  differ  from  OPi'ivt-s  except  in  the  way  they  are  scheduled.  Type  commanders 
determine  test  scheduling  (Department  of  the  Navy,  1986b). 


During  a  scheduled  evaluation,  persons  in  propulsion  ratings  are  evaluated  to 
determine  if  their  job  knowledge  levels  are  sufficient.  This  involves  the  multiple-choice 
test  and  oral  interview,  both  of  which  are  scored  as  pass  or  fail.  The  PEB  examiner  also 
observes  each  watchstation  through  a  number  of  simulated  casualty-control  drills.  The 
examiner  looks  for  safety  violations,  invalid  procedures,  and  loss  of  plant  control.  The 
drill  scenarios  are  based  on  team  performance  and  are  scored  satisfactory  or  unsatis¬ 
factory. 

A  message  report  provides  an  overall  evaluation  of  the  propulsion  plant  readiness 
culminating  in  either  a  satisfactory  or  unsatisfactory  rating.  Major  examination  areas  are 
(1)  operations  (tasks,  drills,  and  evolutions),  (2)  level  of  knowledge,  (3)  fire  fighting 
capability,  (4)  engineering  department  administration,  and  (5)  material  readiness. 

Access  Procedures.  The  PEB  will  generate  ready-made  random  tests  or  will  tailor- 
make  tests  by  selecting  questions  to  fit  any  given  requirement.  (Contact  the  PEB,  32nd 
Street  Naval  Station,  San  Diego.) 

General  Remarks 

The  task  bank  contains  items  that  are  potentially  useful  in  measuring  job  perfor¬ 
mance  as  a  substitute  for  hands-on  tests.  The  bank  is  very  large,  containing  approxi¬ 
mately  3,000  tasks.  There  is  relatively  easy  access  to  the  tasks,  that  is,  tests  can  be 
generated  quickly  and  ma\  be  grouped  according  to  selected  performance  categories. 
However,  the  criticality  and  representativeness  of  the  items  in  the  task  bank  have  not 
been  determined.  This  step  must  be  accomplished  prior  to  using  the  items. 

The  LOE,  OPPRE,  and  OPPE  include  hands-on  test  items.  Hands-on  test  items  are 
also  required  for  JPM.  However,  the  PEB  hands-on  test  items  would  require  major 
modification  for  use.  The  items  do  not  always  clearly  define  the  steps  for  completion; 
steps  would  need  to  be  delineated. 

The  PEB  test  scores  are  not  useful  for  job  performance  measures  due  to  lack  of 
standardization.  The  scoring  procedures  vary  according  to  the  administrator.  Variations 
in  scoring  procedures  prevent  the  use  of  PEB  scores  for  3PM. 


TRAINING  SIMULATORS 


Simulators  are  used  for  a  wide  variety  of  purposes  in  t  s  Navy.  However,  prior  to 

the  recent  development  of  a  simulator  data  base  (M  Best,  &  McDaniel,  1985), 

information  about  the  simulators  used  by  different  g  was  not  readily  available. 

Information  was  collected  from  the  following  ratings:  !  ,  RM,  FC,  and  OS.  A  data 

base  has  been  developed  that  contains  information  abc  simulators  used  in  each  of 

these  ratings. 


It  may  be  feasible  to  use  training  simulators  to  rm  job  performance.  Training 

simulators  provide  relatively  high  fidelity  at  a  lower  than  hands-on  measures. 

Training  simulator  development  and  job  performance  m<.  re  development  strategies 
employ  the  same  underlying  methodology:  job  task  analysis.  Although  the  purposes  vary, 
the  process  uses  much  of  the  same  information  for  formulating  training  objectives  and 
formulating  measures  of  job  performance.  This  factor  leads  to  consideration  of  using 
training  simulators  as  job  performance  measures. 


Importance  to  the  JPM  Project 


Training  simulators  vary  significantly  in  hardware,  software,  output,  and  fidelity. 
This  section  presents  an  overview  of  selected  simulators.  Each  simulator  is  examined  for 
its  potential  use  as  a  measure  of  job  performance.  Simulators  with  hardware  and/or 
software  that  can  be  easily  modified  can  often  be  adapted  for  use  as  job  performance 
measures.  The  modification  of  existing  simulators  offers  substantial  monetary  savings  to 
the  devel  ipers  of  performance  measures. 


Format  of  Training  Simulator  Examination  and  Evaluation 


Training  simulators  examined  in  this  section  include: 

A.  AC  Two  Speed  Motor  Controller  Trainer  (11E15) 

B.  Automated  communications  emulator  (ACE) 

C.  Close  in  weapons  system  (CIWS)  trainer  (11G2) 

D.  Cutler-1  lammer  logic  controller  elevator  trainer 

E.  Electronic  equipment  maintenance  trainer  (EEMT) 

F.  Navy  tactical  data  system  laboratory  (NTDS) 

G.  Practical  applications  laboratory  (PRAC  DECK) 

H.  Small  craft  electrical  system  trainer  (SMALCRAF) 

Each  of  the  above  simulators  is  reviewed  as  follows: 


1.  Hardware  and  software 

2.  Output 

3.  Fidelity 

a.  Hands-on 

b.  Cognitive 

c.  Environmental 


it.  Current  Usage 
5.  Additional  Uses 


1.  Hardware  and  Software.  The  hardware  that  comprises  the  simulator  is  cited.  In 
many  cases  this  description  includes  the  device(s)  that  is  emulated.  Where  applicable,  the 
existing  software  is  described. 

2.  Output.  The  output  category  refers  to  the  output  provided  by  the  simulator  and 
output  consistently  p'  'ided  by  an  observer. 

3.  Fidelity.  delity  of  the  simulator  is  examined.  This  category  is  divided  into 

three  categories.  First,  hands-on  fidelity  is  examined  (a).  Hands-on  fidelity  refers  to  the 
extent  of  similarity  in  the  physical  action  required  when  using  a  simulator  and  actual 
equipment.  Second,  cognitive  fidelity  is  examined  (b).  Cognitive  fidelity  refers  to  the 
extent  of  similarity  in  the  cognitive  processes  required  when  using  the  simulator  and 
actual  equipment.  Third,  environmental  fidelity  is  examined  (c).  Environmental  fidelity 
refers  to  the  extent  of  similarity  in  the  simulator  environment  and  the  actual  equipment 
environment.  All  types  of  fidelity  are  estimated  for  each  device. 

it.  Current  Usage.  Current  usage  describes  for  what  purpose  and  by  whom  the 
device  is  used  at  the  present. 

5.  Additional  Uses.  The  additional  uses  category  contains  suggestions  for  3PM  uses 
of  the  simulator.  This  category  also  contains  suggestions  for  modifications  that  would 
enhance  the  value  of  the  simulator  as  a  performance  measurement  tool. 

AC  Two  Speed  Motor  Controller  Trainer  (HE  13) 

The  HE  15  device  was  designed  to  assist  in  teaching  troubleshooting  of  motor 
controllers. 

Hardware  and  Software 


The  device  is  a  three-dimensional  unit.  It  simulates  an  AC  motor,  motor  controller, 
multimeter,  megger,  and  test  leads.  The  simulator  components  include  a  control  panel, 
computer,  and  power  supply.  Software  has  been  developed  to  represent  29  faults.  The 
software  may  be  used  for  practice  or  testing.  The  practice  mode  presents  faults 
randomly.  In  the  test  mode,  the  rater  selects  faults  that  the  trainee  must  locate.  Actual 
test  equipment  is  not  used;  measurements  made  with  test  equipment  are  simulated. 

Output 

The  software  is  designed  to  record  the  number  of  incorrect  responses. 

Fidelity 

The  hands-on  fidelity  of  the  motor  controller  is  moderately  high  The  equipment  is 
similar;  however,  access  may  be  more  difficult  in  the  Fleet. 

The  cognitive  fidelity  of  the  motor  controller  is  high.  The  required  knowledge  and 
cognitive  processes  related  to  troubleshooting  procedures  are  similar  when  using  the 
simulator  and  in  the  Fleet. 


The  environmental  fidelity  of  the  motor  controller  is  moderate.  In  the  Fleet,  the 
equipment  is  located  in  smaller  quarters.  Also,  the  noise  and  motion  are  greater  in  the 
Fleet. 


Current  Usage 


The  trainer  is  currently  available  for  use  in  training  it  is  not  an  integral  part  of 
training  for  all  EMs. 

Additional  Uses 


The  trainer  can  be  used  for  testing  the  job  perform  >f  EMs  at  the  E4,  E5,  and  E6 

levels.  In  order  to  be  tested  on  the  trainer,  the  EM  w  need  Fleet  experience  with 

motor  controllers  and  a  brief  lesson  in  using  the  trainer.  *  ne  trainer  was  designed  as  a 
generic  device.  Software  could  be  developed  and/or  modified  to  measure  performance  on 
any  motor  controller. 


Automated  Communications  Emulator  (ACE) 


ACE  is  designed  to  simulate  the  functions  of  an  LDMX/NAVCOMPARS  computer. 
ACE  allows  a  user  to  access  data  files  and  to  perform  LDMX/NAVCOMPARS  inrouter, 
router  and  service  clerk  duties. 

Hardware  and  Software 


The  equipment  used  in  the  ACE  includes  a  Zenith  120  desktop  computer,  a  Winchester 
hard  disc  drive,  floppy  disc  drive,  keyboard,  cathode  ray  terminal  (CRT),  and  printer. 
Software  programs  allow  the  use  of  two  modes.  First,  a  review  and  progress  evaluation 
mode  consists  of  the  operating  system,  utility  programs,  application  programs,  data  file 
sets,  and  menus.  From  the  menu,  users  choose  a  section  for  review.  Following  the  review 
period,  the  evaluator  accesses  the  progress  evaluation  phase.  The  user's  knowledge  is  then 
tested.  Second,  the  emulation  mode  consists  of  message  center,  service  center,  and 
LDMX/NAVCOMPARS  operation  scenarios. 

Output 


The  device  is  not  programmed  to  record  user  performance. 

Fidelity 

Hands-on  fidelity  is  very  high.  The  ACE  is  considered  identical  to  Fleet  equipment 
except  for  printer  differences. 

Cognitive  fidelity  is  also  very  high.  The  knowledge  required  to  perform  functions  is 
identical  except  for  printer  operations. 

The  environmental  fidelity  is  considered  to  be  as  high  as  possible.  According  to 
SMEs,  there  are  no  perceivable  differences  between  the  ACE  and  Fleet  environment. 


Current  Usage 

The  ACE  is  used  for  training  RMs. 


Additional  Uses 


Software  modifications  would  allow  the  ACE  to  be  used  to  measure  RM  job 
performance  on  many  router,  inrouter  and  service  clerk  tasks.  Software  modifications 
would  be  necessary  to  ensure  inclusion  of  representative  problems  and  the  recording  of 
performance.  The  device  and  existing  software  programs  are  applicable  to  RMs. 
Extensive  software  development  would  be  required  for  use  with  other  ratings. 

Close  in  Weapons  System  Trainer  ( 1 1 C 2) 

The  1 1 G 2  is  a  :  e-dimensional  training  device  used  by  FCs.  It  is  programmed  to 
teach  troubleshooting  and  maintenance  checking  of  the  C1WS. 

Hardware  and  Software 

The  appearance  and  operation  of  the  11G2  are  similar  to  the  MK-15  Phalanx  C1WS. 
The  11G2  is  computer-controlled  using  software  programs.  Programs  allow  a  user  to 
isolate  and  repair  various  faults  and  perform  maintenance  checks  in  the  electronic  and 
hydraulic  system.  The  11G2  consists  of  panels  with  partial  views  of  all  portions  of  the 
CIWS,  excluding  the  gun.  Users  are  provided  with  information  about  a  problem  or 
maintenance  check.  The  user  then  selects  a  portion  of  the  system  to  view.  This  section 
appears  on  a  panel.  The  user  may  use  test  equipment  and  continue  in  the  process  until 
reaching  the  lowest  possible  unit. 

Output 

The  1 1G2  automatically  logs  the  equipment  conditions,  user  responses,  and  time  for 
problem  completion  for  each  exercise. 

Fidelity 

The  1 1 G 2  is  moderate  in  hands-on  fidelity.  In  some  instances,  repairs  of  the  11G2 
require  less  manual  dexterity  than  repairs  of  the  actual  equipment. 

Cognitive  fidelity  is  very  high.  There  are  few  or  no  differences  in  the  knowledge 
required  when  working  with  the  11G2  and  the  CIWS. 

Environmental  fidelity  is  relatively  low.  The  environment  of  the  11G2  is  very  stable 
and  sanitized  as  compared  to  the  CIWS.  The  CIWS  includes  the  positioning  of  the  system 
in  cramped  quarters,  the  pitching  motion  of  the  ship,  and  additional  safety  considerations. 

Current  Usage 

The  11G2  is  used  by  FCs  as  a  training  component.  All  students  are  exposed  to  this 
device. 

Additional  Uses 

The  1 1 G 2  was  designed  as  a  specific  trainer.  All  aspects  of  the  system,  including 
appearance,  are  based  on  the  MK-15  Phalanx  CIWS.  The  I1G2  can  be  utilized  for 
measurement  of  job  performance  on  this  CIWS.  It  would  be  difficult  to  adapt  the 
equipment  for  performance  measurement  of  other  devices. 
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Cutler-Hammer  Logic  Controller  Elevator  Trainer 


The  Cutler-Hammer  logic  controller  elevator  trainer  is  designed  to  simulate  faults  in 
a  logic  controller.  It  teaches  troubleshooting  to  the  lowest  possible  component. 

Hardware  and  Software 


The  device  consists  of  several  items  mounted  on  a  table.  These  items  include  a 
problem  simulator,  a  flat  car  containing  the  remote  devices  found  in  actual  elevators,  and 
a  logic  controller.  The  trainer  is  used  in  the  following  manner.  The  rater  inserts  one  or 
more  faults.  The  trainee  uses  schematics  and  test  equipment  to  troubleshoot  the 
equipment.  The  task  is  completed  when  the  trainee  locates  the  fault(s). 

Output 

Information  is  not  recorded  by  the  elevator  trainer.  The  rater  records  the  number  of 
errors  and  time  for  completion  on  a  standardized  checklist. 

Fidelity 

The  hands-on  fidelity  is  low.  The  trainer  represents  outdated  equipment;  many 
factors  have  changed.  The  trainer  simulates  fewer  levels  than  are  actually  required  in 
practice. 

The  cognitive  fidelity  is  moderately  high.  The  cognitive  processes  required  to 
troubleshoot  the  trainer  are  similar  to  the  processes  required  to  troubleshoot  in  the  Fleet. 
However,  faults  that  are  important  in  the  Fleet  are  not  represented  on  the  trainer. 

The  environmental  fidelity  is  low.  First,  the  trainer  places  the  logic  console  and 
elevator  in  close  proximity.  In  the  Fleet,  the  logic  console  and  elevator  are  not  located  in 
the  same  area;  thus,  repairs  are  more  difficult.  Second,  the  trainer  allows  easy  access  to 
equipment.  Access  is  much  more  difficult  on  board  ship. 

Current  Usage 

The  elevator  trainer  is  designed  for  use  by  EMs  for  training;  however,  its  current  use 
s  very  limited. 

Additional  Uses 


The  Cutler-Hammer  logic  controller  elevator  trainer  is  designed  for  a  very  specific 
purpose,  troubleshooting  faults  in  logic  controllers.  The  trainer  is  also  designed  for  a  very 
specific  group,  EMs  with  extensive  Fleet  experience  and  training  in  the  use  of  the  device. 
The  device  could  be  updated  for  use  in  measuring  job  performance  for  troubleshooting 
logic  controllers.  Due  to  the  specificity  of  the  devices,  it  serves  an  extremely  limited 
population. 

Electronic  Equipment  Maintenance  Trainer  (EEMT) 

The  EEMT  is  a  training  simulator  used  for  teaching  maintenance  procedures  and 
troubleshooting  tasks  on  electronics  equipment. 
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Hardware  and  Software 


The  EEMT  consists  of  a  microcomputer,  video  display  with  touch  sensitive  panels,  and 
a  videodisc  player.  It  is  a  two-dimensional  device.  Software  programs  have  been  created 
for  use  with  the  unit.  Structured  and  free  play  problems  are  included.  A  structured 
program  provides  a  lesson  using  a  directive  interactive  format.  Free  play  programs  allow 
the  trainee  to  follow  many  different  paths  in  solving  the  problem.  Each  program  contains 
a  series  of  problems  s;sociated  with  a  specific  piece  of  electronics  equipment.  Menus, 
text,  problems,  and  equipment  images  {from  a  photographic  data  base)  are  presented  on 
the  video  display.  Trainees  interact  with  the  unit  by  hitting  a  touch  panel  on  the  video 
display. 

Output 

The  number  of  correct  and  incorrect  responses  are  automatically  recorded. 

Fidelity 

The  EEMT  is  relatively  low  in  hands-on  fidelity.  There  is  little  similarity  between 
the  physical  characteristics  of  the  EEMT  and  the  actual  equipment. 

The  cognitive  fidelity  is  high.  The  cognitive  processes  required  for  simulated  repair 
and  maintenance  of  equipment  are  similar  to  the  processes  when  repairing  actual 
equipment. 

The  environmental  fidelity  is  low.  The  EEMTs  are  housed  in  a  relatively  comfortable 
environment;  factors  such  as  noise,  crowding,  and  motion  are  not  included  in  the 
classroom. 

Current  Usage 

At  this  time,  the  EEMT  is  used  by  two  ratings,  ETs  and  FCs.  ETs  use  the  EEMT  for 
remediation;  it  is  not  currently  part  of  the  core  curriculum.  Data  bases  have  been 
developed  for  several  types  of  electronic  equipment.  FCs  use  the  EEMT  as  an  integral 
aspect  of  their  training  program.  The  EEMT  provides  FCs  with  training  for  troubleshoot¬ 
ing  on  the  11D8A  Radar  Trainer.  First  termers  are  the  primary  users  of  the  device. 

Additional  Uses 

The  EEMT  is  a  generic  trainer  that  can  be  used  for  performance  testing.  Software 
can  be  developed  for  training  and/or  testing  on  most  pieces  of  equipment.  Software  may 
also  be  modified  to  provide  more  job  performance  data.  The  EEMT  is  user  friendly;  it  is 
very  easy  to  learn  to  use  the  simulator. 

Navy  Tactical  Data  System  Laboratory  (NTDS) 

The  NTDS  laboratory  simulates  an  operational  NTDS.  The  laboratory  is  designed  to 
train  personnel  to  access  and  enter  data,  as  required  for  tactical  maneuvers. 

Hardware  and  Software 


The  NTDS  laboratory  is  a  three-dimensional  trainer  that  provides  high-fidelity 
simulations.  It  includes  the  consoles  used  in  the  Fleet  and  data  entry  panels  that  are 


similar  to  those  used  in  the  Fleet.  The  software  contains  scenarios  that  are  representa¬ 
tive  of  situations  that  occur  in  the  Fleet. 


Output 


Performance  is  automatically  measured  by  the  simulator.  All  nonverbal  responses 
can  be  recorded;  however,  a  limited  output  is  currently  received. 

Fidelity 

The  hands-on  fidelity  is  high.  The  equipment  is  identical  to  the  equipment  used  in  the 
Fleet. 

The  cognitive  fidelity  is  high.  The  cognitive  processes  required  to  operate  the 
training  equipment  and  actual  equipment  are  identical. 

The  environmental  fidelity  is  high.  The  laboratory  is  designed  to  be  similar  to  actual 
surroundings.  The  motion  of  the  ship  is  absent  in  the  laboratory. 

Current  Usage 

The  laboratory  is  an  integral  part  of  training  for  NTDS. 

Additional  Uses 


The  NTDS  laboratory  may  be  used  to  measure  job  performance  of  OSs.  The 
laboratory  includes  various  sensors,  processor  consoles,  and  tactical  programs.  SMEs 
estimate  that  the  Jab  covers  approximately  25  percent  of  an  OS's  job  tasks.  The 
laboratory  could  effectively  measure  these  tasks  and  similar  tasks  for  any  rating. 
Extensive  hardware  and  software  changes  would  be  required  to  include  other  tasks. 

Practical  Applications  Laboratory  (PRAC  DECK) 

The  PRAC  DECK  laboratory  simulates  afloat  and  ashore  communications  centers. 
The  PRAC  DECK  contains  equipment  used  to  transmit  and  receive  messages.  The 
equipment  is  similar  to  the  equipment  found  in  a  Fleet  radio  shack. 

Hardware  and  Software 


The  PRAC  DECK  contains  nearly  identical  equipment  to  that  found  in  a  Fleet  radio 
shack.  The  equipment  may  be  a  generation  older;  however,  it  is  functionally  similar. 
Specifically,  the  PRAC  DECK  contains  the  following: 


AN/WRC-1,  HF  transmitter 
AN/URT-23V,  HF  transmitter 
R-1051,  HF  receiver 
AN/URC-9,  UHF  transceiver 
AN/UGC-6,  teletypewriter  (send) 
AN/UGC=25A,  teletypewriter  (receive) 
KWR-37/TSEC,  cryptographic  equipment 
KG-14/TSEC,  cryptographic  equipment 
KW-7/TSEC,  cryptographic  equipment 
KY-75/TSEC,  cryptographic  equipment 
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KY-S/TSEC,  cryptographic  equipment 

KY-5S/TSEC,  cryptographic  equipment 

KWX-S/TSEC,  cryptographic  remote  phasing  unit 

SB-9SS/SRT,  transmitter  transfer  switchboard 

AN/URA-17,  converter/comparator  group 

CV2460/SCG,  telegraph-telephone  signal  converter 

AN/SRA-12-3/UG,  patch  panel 

SB973/SRR,  patch  panel 

SB863/SRT,  patch  panel 

SB1203/UG,  patch  panel 

SB1210/UGQ,  patch  panel 

SB-3195,  patch  panel 

C10315,  remote  switching  control  matrix 

C1004B,  transmitter-teletypewriter  control 

TA-970/U,  secure  phone 

Output 

The  PRAC  DECK  includes  operational  equipment  fed  into  dummy  loads.  The  primary 
form  of  output  is  a  hard  copy  of  the  message  that  is  sent  or  received.  The  messages  may 
be  checked  for  accuracy.  In  addition,  human  raters  monitor  the  ratees  performance. 

Fidelity 

The  hands-on  fidelity  is  very  high.  It  is  less  than  perfect  because  some  pieces  of 
equipment  are  not  the  most  current  models. 

The  cognitive  fidelity  is  very  high.  The  knowledge  required  to  function  in  a  PRAC 
DECK  is  almost  identical  to  the  knowledge  required  to  function  in  an  afloat  or  ashore 
communications  center. 

The  environmental  fidelity  is  moderate.  The  PRAC  DECK  is  more  spacious  than 
some  centers.  It  is  also  free  of  static,  which  is  a  problem  in  centers. 

Current  Usage 

The  PRAC  DECK  is  used  for  training  of  RMs.  It  provides  a  working  environment  for 
performing  nine  RM  job  positions. 


Additional  Uses 

The  PRAC  DECK  is  a  highly  specialized  environment.  It  can  be  used  to  measure  the 
performance  of  RM.  However,  the  essence  of  the  PRAC  DECK  allows  it  to  be  used  only 
by  ratings  that  are  familiar  with  communications  equipment.  Users  of  the  PRAC  DECK 
are  required  to  possess  a  security  clearance. 

Small  Craft  Electrical  System  Trainer  (HE  18) 

The  11E18  simulates  the  operation  of  a  small  craft  electrical  system.  This  device  is 
designed  to  teach  troubleshooting  to  EMs  by  presenting  faults  in  several  pieces  of 
equipment. 


Hardware  and  Software 


The  I1E18  resembles  a  work  bench  on  which  equipment  is  mounted.  The  unit  is  based 
on  a  control  panel  attached  to  a  computer  and  power  supply.  The  components  on  the 
bench  include  a  starter,  an  alternator,  a  battery,  a  regulator,  and  a  lighting  system.  The 
software  consists  of  fixed  programs  that  allow  the  user  to  isolate  and  repair  one  or  more 
of  the  12  possible  faults.  The  trainer  selects  the  fault(s)  to  be  used  and  then  the  user 
troubleshoots  by  accessing  information  and  determining  the  required  repair.  The  user 
enters  a  repair  choice  on  the  keyboard;  lights  flash  to  indicate  a  correct  or  an  incorrect 
choice. 


Output 


Correct  and  incorrect  choices  are  recorded  cumulatively  and  by  percentage. 


Fidelity 

The  hands-on  fidelity  is  low.  The  11E18  is  organized  quite  differently  than  a  ship  is 
organized.  In  addition,  standard  Fleet  troubleshooting  equipment  cannot  be  used  on  the 
simulator  due  to  its  design. 

The  cognitive  fidelity  is  low.  The  programmed  faults  are  not  representative  of  the 
faults  found  on  ships.  5MEs  considered  5  of  the  12  faults  to  be  too  simplistic. 

The  environmental  fidelity  is  low.  The  Fleet  environment  contains  many  elements 
that  are  not  present  in  the  11E18,  such  as  motion  and  difficult  access  to  equipment. 

Current  Usage 


The  11E18  is  currently  used  as  a  trainer  for  EMs.  Individuals  can  be  trained  to  use 
the  machine  in  approximately  I  to  3  hours. 


Additional  Uses 

The  device  could  be  used  to  measure  job  performance;  however,  several  changes 
would  first  need  to  be  made.  There  is  a  need  for  increased  machine  reliability;  several 
faults  were  not  available.  The  relevant  faults  could  be  added.  Different  or  additional 
equipment  could  be  included  on  the  work  bench  to  fit  the  population  in  question. 


RECOMMENDATIONS 

The  Navy  currently  holds  a  great  deal  of  information  that  is  pertinent  to  3PM. 
Although  no  existing  data  bases,  programs,  or  simulators  can  be  used  as  complete 
performance  measures,  each  of  these  systems  may  support  the  development  of  job 
performance  measures. 

Data  Bases 

NOTAP  provides  a  comprehensive  data  base  consisting  of  task  lists,  encompassing 
approximately  two-thirds  of  all  Navy  ratings.  These  data  are  collected  in  a  questionnaire 
format.  It  is  recommended  that  the  NOTAP  data  be  used  as  a  starting  point  in  the 
development  of  job  performance  measures.  The  NOTAP  questionnaires  could  be  used  for 


gathering  SMEs'  judgments  on  tasks  to  include  in  a  3PM  program.  SMEs  could  be  asked  to 
rate  the  criticality  of  tasks  on  the  NOTAP  questionnaire.  These  criticality  ratings  could 
then  be  used  to  select  tasks  for  performance  measurement.  NEPD1S  is  limited  in  use 
because  it  has  been  completed  for  only  one  rating  and  has  not  been  maintained. 

Programs 

The  PQS  system  is  designed  to  provide  standardized  information  regarding  the 
knowledge  and  duties  associated  with  each  watchstation.  Each  PQS  refers  to  a  particular 
watchstation;  approximately  700  PQS  have  been  developed.  PQS  may  be  useful  in 
formulating  job  performance  measures.  Each  PQS  booklet  is  the  result  of  a  task  analysis 
for  a  particular  watchstation.  The  results  of  the  task  analysis  are  presented  in  question 
and  answer  format.  It  is  difficult  to  use  PQS  because  the  majority  of  items  do  not  include 
the  steps  for  task  completion;  however,  some  of  these  items  may  be  translated  into 
performance  test  questions.  The  results  of  PQS  are  not  useful  for  job  performance  testing 
due  to  scoring  procedures.  PQS  is  scored  by  a  variety  of  supervisors  who  may  vary  in 
testing  style  and  interpretation  of  performance.  It  is  recommended  that  PQS  be  used  as  a 
developmental  tool  in  the  job  performance  test  construction  phase.  SPPOT  uses  EOSS  and 
PQS  for  development  of  job  performance  and  hands-on  items.  The  utilization  of  technical 
information,  such  as  EOSS,  may  prove  to  be  beneficial  in  the  construction  of  performance 
tests. 

The  FFS  provides  an  evaluation  of  Navy  training  programs.  This  system  offers 
valuable  information  to  the  designers  and  trainers;  it  does  not  provide  job  performance 
information.  The  system  offers  training  course  information;  it  does  not  offer  job 
performance  information.  However,  the  3PM  program  may  benefit  from  a  strategy  used 
by  this  system;  that  is,  3PM  can  also  use  the  Catalog  of  Navy  Training  Courses  to  gather 
technical  information.  In  addition,  the  3PM  program  can  assist  the  FFS.  3PM  acquires 
data  that  are  equivalent  to  the  FFS's  Level  3  data.  Level  3  data  can  be  used  by  the  FFS's 
group  to  confirm  the  Level  2  data. 

The  PTEP  is  a  method  of  evaluating  training  in  specified  ratings.  The  program  is 
advantageous  due  to  its  rigorous  development.  A  thorough  task  analysis  is  conducted  and 
updated,  and  all  tests  are  criterion-referenced.  PTEP  may  be  of  limited  use  to  a  job 
performance  program  due  to  its  limited  scope  and  high  security.  PTEP  has  been 
developed  for  only  five  ratings.  SP-15  must  authorize  all  use  of  PTEP  materials.  The 
security  of  PTEP  makes  it  difficult  to  consider  for  use  in  the  development  job 
performance  measures. 

The  PEB  evaluates  the  safety  of  a  ship's  steam  propulsion  plant  and  the  readiness  of 
the  watch  teams.  The  task  bank  of  PEB  contains  a  large  number  of  items  that  can  useful 
in  developing  performance  measures.  The  task  bank  items  may  serve  as  a  basis  for 
gathering  5ME  judgments  regarding  the  representativeness  and  criticality  of  potential 
paper-and-pencil  test  items.  In  a  similar  manner,  PEB  hands-on  items  may  also  be  used  as 
a  basis  for  development  of  hands-on  job  performance  measures. 

Simulators 

Simulators  may  be  used  as  substitute  measures  of  hands-on  job  performance.  One 
device,  the  small  craft  electrical  trainer,  can  be  eliminated  from  consideration  because 
extensive,  costly  modifications  would  be  required. 


23 


Several  devices  have  been  developed  for  use  with  a  single  rating.  For  example,  the 
AC  two-speed  motor  controller  trainer  and  the  Cutler-Hammer  logic  controller  elevator 
trainer  are  designed  for  use  by  the  EMs.  The  ACE  is  designed  for  use  by  the  RM  rating. 
The  C1WS  trainer  is  designed  for  use  by  FCs.  With  hardware  and/or  software  modifica¬ 
tions,  each  of  these  devices  could  be  used  for  performance  measurement  of  the  rating  for 
which  it  was  designed.  However,  extensive  hardware  and  software  modifications  would  be 
required  for  use  with  any  other  rating. 

Two  devices  are  full  mock-ups:  the  PRAC  DECK  and  the  NTDS  laboratory.  Each 
contains  many  pieces  of  actual  equipment,  resulting  in  very  high  fidelity  for  the  PRAC 
DECK  and  NTDS.  The  PRAC  DECK  can  be  considered  as  a  measure  of  job  performance 
for  ratings  that  work  with  communications  equipment,  specifically  the  RM  rating.  Use  of 
the  PRAC  DECK  is  limited  for  three  reasons:  (1)  a  security  clearance  is  required,  (2)  this 
device  is  used  extensively  at  this  time;  and  (3)  some  equipment  must  be  modified  to 
provide  performance  data.  If  these  three  obstacles  can  be  overcome,  the  PRAC  DECK  is 
potentially  an  excellent  source  of  job  performance  data.  NTDS  is  also  recommended  for 
use.  This  laboratory  contains  actual  equipment  and  identical  replicas  of  actual  equip¬ 
ment.  Software  has  been  developed  to  provide  scenarios  that  are  representative  to  those 
occurring  in  the  Fleet.  Although  the  NTDS  laboratory  has  the  capability  to  record  a  great 
deal  of  information  regarding  user  performance,  the  software  would  need  to  be  modified 
to  record  meaningful  performance  data.  NTDS  is  also  a  potentially  good  source  of  job 
performance  data  for  OSs. 

One  device  with  the  most  potential  as  a  substitute  measure  of  job  performance  is  the 
EEMT.  The  EEMT  was  originally  developed  as  a  generic  trainer.  This  device  is  currently 
used  by  two  ratings,  ETs  and  FCs,  and  could  be  modified  to  accommodate  a  great  number 
of  ratings.  The  EEMT  can  be  modified  with  relative  ease  to  test  troubleshooting  and 
maintenance  tasks  on  equipment.  It  is  recommended  that  the  EEMT  be  used  as  a 
substitute  measure  of  hands-on  job  performance. 


CONCLUSIONS 

The  use  of  data  bases,  programs,  and/or  simulators  for  more  than  one  purpose  avoids 
duplication  of  effort,  thus  reducing  cost  and  time.  If  the  3PM  program  is  expanded 
beyond  the  six  ratings  currently  being  covered,  analyses  of  additional  data  bases, 
programs,  and  simulators  should  be  conducted  to  form  a  board  network  of  job  performance 
information  for  use  by  many  groups. 
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